Neural networks are a class of electronic circuits which emulate higher-order brain functions such as memory, learning and/or perception/recognition. These networks basically adopt a parallel approach to problem solving in which pattern recognition and mapping problems can be solved in thousandths of the time required by a conventional Von-Neuman computer with its single-instruction cycle times. For an introduction to the field of artificial neural networks, reference may be made to an article by Richard P. Lippmann entitled "An Introduction To Computing With Neural Nets", published in IEEE ASSP Magazine, April 1987. That article describes commonly used networks and simple neuron-like components.
In its simplest form, a neural network generally takes the form of a matrix comprising a set of horizontal lines which cross and contact a set of vertical lines. The horizontal lines simulate the function of axons in the cortex of the brain and are used as inputs. The vertical lines simulate the function of dendrites extending from neurons. Each vertical line terminates at a voltage summing device which acts to simulate the function of the neural cell body. These networks are frequently referred to as single layer perceptrons, and have been developed for their ability to learn to recognize simple patterns. Examples of such networks and their computational elements are disclosed in U.S. Pat. Nos. 4,950,917; 4,904,881; 4,956,564; and 4,961,002, all of which are assigned to the assignee of the present application.
In order to solve problems that are not linearly dependent, or not linearly separable, a multi-layer neural network (i.e., a multi-layer perceptron) is required. Multi-layer networks overcome many of the limitations of single-layer networks by including additional layers containing hidden units or nodes that are not directly connected to both the input and output nodes. By way of example, a multi-layer neural network is disclosed in U.S. Pat. No. 4,906,865, which is also assigned to the assignee of the present invention. The number of layers in the network is directly related to the complexity of the problem which the net is capable of solving. For instance, a three-layer perceptron is capable of generating arbitrarily complex decision regions.
In the past, one of the main drawbacks of multi-layer perceptrons has been that each of the nodes associated with the additional layers includes a a corresponding set of neurons. These neurons usually consist of summing and Sigmoid functions. However, because the electrical implementations of neurons tend to be very area intensive, multi-layer networks have proven inefficient and costly to build. Therefore, what is needed is a means of implementing a multi-layer neural network employing a minimum number of neurons.